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The ability to use symbols is the pinnacle of human intelligence, but has yet to be fully replicated in 
machines. Here we argue that the path towards symbolically fluent artificial intelligence (AI) begins 
with a reinterpretation of what symbols are, how they come to exist, and how a system behaves when it 
uses them. We begin by offering an interpretation of symbols as entities whose meaning is established 
by convention. But crucially, something is a symbol only for those who demonstrably and actively par- 
ticipate in this convention. We then outline how this interpretation thematically unifies the behavioural 
traits humans exhibit when they use symbols. This motivates our proposal that the field place a greater 
emphasis on symbolic behaviour rather than particular computational mechanisms inspired by more 
restrictive interpretations of symbols. Finally, we suggest that AI research explore social and cultural 
engagement as a tool to develop the cognitive machinery necessary for symbolic behaviour to emerge. 
This approach will allow for AI to interpret something as symbolic on its own rather than simply manip- 
ulate things that are only symbols to human onlookers, and thus will ultimately lead to AI with more 
human-like symbolic fluency. 
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1. Introduction 


The ability to use symbols makes it possible to reason about the laws of nature, fly to the moon, write 
and recite poetry, and evoke thoughts and feelings in the minds of others. Newell and Simon extolled 
the virtues of symbols in their Turing award winning lecture in 1976, claiming that “[s]ymbols lie at 
the root of intelligent action” and should therefore be a central component in the design of artificial 
intelligence (Newell and Simon, 1976). With this in mind they formulated the physical symbol system 
hypothesis (PSSH), which states that a physical symbol system has the necessary and sufficient means 
for general intelligent action (Newell, 1980; Newell and Simon, 1976). Thus began a decades-long 
research program in AI (often referred to as Good Old-Fashioned AI, or GOFAD) to create intelligence 
by applying the syntactic mechanisms developed in computer science, logic, mathematics, physics, 
and psychology. 


Newell and Simon argued that their hypothesis is empirical, meaning that experimental and 
observational evidence can prove it to be false (Newell and Simon, 1976). In the time since its 
proclamation, there have been demonstrations that human thought is not best explained with formal, 
syntax-centric models (Johnson-Laird, 1999; Kosslyn and Hatfield, 1984; McClelland et al., 2010; 
Rumelhart and McClelland, 1986), and accumulating feats of intelligence that heavily depend on 
non-GOFAI methods (Akkaya et al., 2019; Brown et al., 2020; Mnih et al., 2015; Silver et al., 2016, 
2017). There have also been pointed rational arguments, stating that GOFAI methods alone are 
insufficient because they do not connect to the real world (Harnad, 1990), and philosophical attacks 
on the fundamental assumptions underlying GOFAI methods (Dreyfus, 1972, 1965; Smith, 2019). 


Contemporary AI research strays from GOFAI methods, and instead leverages learning-based 
artificial neural networks (LeCun et al., 2015). Nevertheless, the apparent weaknesses of current 
connectionist-based approaches are leading some to call for a return of symbolic methods, a de- 
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emphasis of connectionist methods, or some reconciliation of the two (Garcez and Lamb, 2020; 
Garnelo and Shanahan, 2019; Lake et al., 2017; Marcus, 2018, 2020). As the argument goes, perhaps 
it is not true that these methods alone lead to human-like intelligent action, but rather, the path 
towards intelligent action lies in a chimeric assembly of old methods with the new. 


However, here we argue that building a system that exhibits human-like engagement with symbols 
is not a necessary consequence of deploying “symbolic methods” (Boden, 2014) (which we term 
GOFAI-like), whether in conjunction with neural networks or not. This is because these approaches 
miss much of the full spectrum of symbolic capabilities that humans exhibit. To that end, we focus 
on describing the key components of symbolic behaviour, which is the observable consequence of 
symbol-use that we are ultimately concerned with, and which can be directly measured and targeted 
when building AI. Indeed, it was behavioral evidence from humans—including their introspection 
about problem solving—that Newell and Simon (1976) cited in support of the PSSH. Our argument 
will center on the question: how does a symbolic thinker behave if they interpret something as a 
symbol, and can these behaviours be replicated in machines? An intriguing aspect of our analysis is 
that the traits associated with symbolic behaviour can be made thematically coherent if we reinterpret 
what symbols are and how they come to exist. To that end, our argument will proceed as follows: 


We will start by presenting this interpretation of symbols, which underpins the broader character- 
istics of symbol-use in humans, and contrasts with narrower interpretations that focus on the syntactic 
manipulation of discrete tokens. In particular, our interpretation emphasizes how symbol meaning 
is established by convention, how symbols only exist with respect to an interpreter that exhibits 
certain behaviours, how symbol meaning is independent of the properties of their substrate, and how 
symbols are situated in broader symbolic frameworks. We will then articulate some behavioral traits 
associated with symbol-use in humans and link them back to our interpretation of symbols. We then 
draw inspiration from developmental psychology to argue for social and cultural engagement as a 
positive pressure for creating AI that can exhibit symbolic behaviour. Finally, we discuss how our 
perspective relates to other views on symbols and AI, and conclude by summarizing our argument 
and the research directions we prescribe. 


2. Properties of Symbols and Symbolic Systems 


Although symbols are conceptually and historically important in the study of AI, they do not have a 
canonical definition in the literature. Working definitions are often implicit, or only vaguely referenced, 
presumably because we can leverage intuitions gained from the study of logic, computer science, 
philosophy, mathematics, or semiotics. But the lack of definition has led to tension and disagreement. 
For example, in a series of articles about the first self-driving car, ALVINN (Pomerleau, 1989), Touretsky 
and Pomerleau expressed frustration that seemingly anything that “designates” or “denotes” is raised 
to the status of “symbol” by some researchers, rendering the concept operationally vacuous (Touretzky 
and Pomerleau, 1994; Vera and Simon, 1993). 


Newell and Simon define symbols as a set of interrelated “physical patterns” that could “designate 
any expression whatsoever” (Newell and Simon, 1976). Indeed, the property of arbitrary designation 
is a common theme throughout the literature. But, crucially, Newell and Simon do not explicitly state 
who creates or is aware of the designation. What we hope to show is that considering the answer to 
“who creates or is aware of the designation” is fundamental for creating AI with symbolic competence. 


First, we offer a disclaimer. Our goal is not to proclaim that any particular definition is objectively 
correct. Rather, different uses of the word “symbol” offer unique perspectives. We hope to offer 
guidelines for thinking about symbols that offer promise for building AI that can achieve what we 
describe as “symbolic behavior”, and further, to argue that achieving symbolic behavior is a useful 
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goal. For ease of writing, we will speak declaratively in the remainder of the paper (e.g., “symbols 
are...”), but we do not wish to reject the validity of other perspectives. Similarly, we will use terms 
like “understanding,” and “meaning” that have acquired a conventional meaning but are difficult to 
define precisely. While this vagueness is unavoidable in some of the issues we must discuss, we will 
attempt to describe our proposed path forward concretely in terms of operationalizable behavioral 
criteria and research directions that stem from our interpretation of a symbol. 


A symbol consists of some substrate—a scratch on a page, an air vibration, or an electrical pulse— 
that is imbued with meaning by matter of convention. However, the substrate is only a symbol to those 
who demonstrably: (i) understand the established conventional meaning, and (ii) understand that 
the meaning is what it is because of convention. That symbols’ meanings are a matter of convention 
aligns with Newell and Simon’s criterion of arbitrary designation. But we extend this criterion by 
emphasizing the role of the interpreter who understands the conventional relationship between a 
substrate and what it represents. A symbol’s meaning is therefore not intrinsic to its substrate, nor 
is it objective. Symbols are subjective entities; to those who do not participate in the established 
convention, a scratch on a page is simply a scratch on a page (or at best is recognizable as the substrate 
of a symbol, but is devoid of meaning, as is a character from an alphabet we do not understand). 


Our definition of a symbol draws on the work of the philosopher Charles Sanders Peirce. Peirce 
outlined three categories of relation—icons, indices, and symbols—whose definitions illuminate the 
role of convention in establishing meaning (Deacon, 1998; Hartshorne et al., 1931). According to 
Peirce, icons make reference by way of similarity. A sculpture of an elephant bears literal physical 
resemblance to a real elephant, and hence is iconic. Indices, on the other hand, leverage some 
temporal or physical connection to the things to which they refer. The mercury in a thermometer, for 
example, changes its height in accordance with the temperature. But to Peirce, symbols depend on an 
“agreed upon link”, regardless of the actual physical or temporal characteristics of the medium. It is 
only by agreement that a flag comes to symbolize one country rather than another. There is nothing 
inherent in the properties of cotton or ink that makes a flag symbolize Canada. The delineation 
between icons, indices, and symbols need not be sharp, and in reality Peirce’s proposed categorization 
is probably best understood as a guideline. Nevertheless, for the purposes of our argument we can 
entertain an idealized notion of a symbol wherein meaning is established purely by convention. 


There is much to unpack in this interpretation of symbols. We’ll first describe the relationship 
between representation and meaning as determined by convention, and in the later section titled 
“Symbolic Behaviour” we’ll discuss the role of the symbolic interpreter, what it means to understand 
meaning established by convention, and what it means to understand that meaning is what it is 
because of convention. 


2.1. The Consequences of Construing Symbol Meaning as Established by Convention 


A first consequence to construing symbol meaning as a matter of convention (an “agreed upon link”) is 
that a symbol’s meaning is rendered independent of the properties of the symbol’s substrate. When a 
representation’s properties do not convey meaning in and of themselves, representations can come to 
be used to symbolize anything!. And similarly, the same meaning can be ascribed to representations 
with wildly different characteristics. For example, consider the differences between written text and 
oral speech. The underlying meaning can be similar, even though the mediums are distinct. The 
converse is also possible: similar representations can symbolize very different things to different 
interpreters (such as a cross-lingual homophone). This connects with the “arbitrariness” emphasized 


1To emphasize this point by taking it to the extreme, as computer scientists are well aware even a single bit (i.e. 0 or 1) 
can come to symbolize an elephant, or democracy, or your state of mind when you visited a grocer and they were out of 
oranges. 
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by (Newell and Simon, 1976), but with a different focus—it is important to speak about symbols with 
reference to a particular interpreter who generates a particular interpretation, and to not speak about 
symbols as if they transmit objective meanings to any possible interpreter. 


Shannon’s communication theory exploits the fact that material substrates do not inherently 
symbolize anything (Shannon, 1948). Once one establishes a coding scheme, then and only then 
do sequences of electrical pulses, for example, come to be about something (i.e., they symbolize 
something). But the pulses are only symbolic to those privy to the coding scheme. Part of the insight 
behind Shannon’s theory is that it measures quantities (e.g. entropy) based on properties of a symbol’s 
substrate (e.g. an electrical pulse), and not based on what the arbitrary meaning of a message happens 
to be. In other words, Shannon realized that we can communicate symbols efficiently because we can 
exploit the fact that symbol meaning is imposed, and hence because we can indiscriminately factor 
meaning out of the equation when we shuttle signals around. Crucially, to a telephone, or an electrical 
cable, or a drum, electrical pulses do not mean, nor symbolize, anything. And there is nothing they 
can glean from the physical properties of these pulses to make them mean what they do for humans. 


A second consequence of construing symbol meaning as a matter of convention, therefore, is 
that something can be a symbol from the perspective of one system, but not another. Turning to 
communication theory again, suppose Bob establishes a coding scheme with Alice but not with Sally. 
The sequences of electrical pulses that Bob sends Alice are symbolic to Alice (that is, they have an 
associated meaning based on an agreed convention). But to Sally, they are just electrical pulses. The 
only meaning Sally can hope to glean from the message must be extracted from the literal, physical 
properties of electricity (such as its charge), or the temporal electric signal (such as its entropy), or 
from some complicated inference process wherein she predicts the coding scheme using privileged, 
outside knowledge. Indeed, this fact is the base upon which open, encrypted communication is built. 


A third consequence of construing symbol meaning as convention is that we can situate symbols 
in a broader symbolic system, wherein symbols refer to other symbols in the system in addition to 
referring to entities in the world. This property is also central to GOFAI-like approaches to symbols. 
Indeed, the consequences of taking this observation to the extreme were popularly explored in Searle’s 
Chinese Room Argument (Searle et al., 1980). Just as in these classical approaches, convention can 
dictate that a symbol’s meaning is tied to the meaning of another symbol. We emphasize that there is 
not simply a unidirectional definition of one symbol by another, but rather that symbol meaning is a 
holistic property of the entire symbol system. Deacon similarly argues that the referential power of 
symbols is derived from their “positions in an organized system of other symbols” (Deacon, 1998). 
So, in characterizing symbols it is often not enough to make reference to a fixed system into which 
they happen to be placed, or to treat symbols as isolated, independent entities. Instead we must 
consider the bi-directional influences caused by a symbol’s placement in a broader symbolic system. 


6, 99 


As an example, consider the concept of a vector in mathematics. One can use the symbol “v” to 
denote an arbitrary example of a vector, such as (0.2, -0.5, 0.7) for 3-dimensional Euclidean space. 
But a vector is more than a symbolic pointer to a collection of numbers. What vectors are, generally, 
is determined by the notion of a vector space, which is an agreed upon set of axioms and rules that 
describe a grander symbolic framework within which vectors live. So, describing a vector as simply 
an ordered collection of elements is in many ways impoverished. For example, ordered collections do 
not tell us anything about the rules used to combine or shape vectors. Notably, the axioms and rules 
that describe the grander symbolic system are also symbolic, creating a webbed relationship wherein 
one type of convention impacts another. 
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3. Symbolic Behaviour 


The previous section offered a view of symbols that emphasized the role of an interpreter. It claimed 
that an entity is not a symbol in any objective sense, but rather, is a symbol for an interpreter who treats 
it as such. We will now ask: how does an interpreter behave if it interprets something as a symbol? If 
we can identify the particular behavioural traits that are consequences of engaging with symbols, 
then we can use them as tangible goals for creating AI that is as symbolically fluent as humans, and 
further, use their presence as evidence that a system interprets something as a symbol. What we 
hope to show is that the behavioural traits exhibited by symbol-users indicate an active participation 
in an infrastructure of meaning by convention, with the accompanying understanding that meaning is 
conventional. In other words, the behavioural traits exhibited by symbol users directly motivate the 
interpretation of symbols provided in the previous section, which paid particular attention to the role 
of convention in establishing symbol meaning. 


We will begin with a few examples. Suppose a symbolic thinker creates a symbol. For example, a 
person can use the word “dax” as a command for their dog to sit. Consequently, their dog might learn 
to sit when it hears the word “dax”. Is the word “dax” a symbol to the dog? Similarly, consider apes 
that learn some simple sign language. Apes can enact the motor program for certain gestures, and 
they can form associations between these gestures and some outcomes. For example, suppose they 
learn that a particular hand movement results in them receiving a treat. There is nothing inherent 
to the motor movement itself that entails that the ape should get a treat. So, the gesture and its 
outcome are more-or-less arbitrarily linked, which aligns with the properties we previously outlined 
about symbols, as well as with the “arbitrariness” highlighted by Newell and Simon (1976). Is the 
gesture a symbol for the ape? 


Now consider an adult human performing the gesture. Humans understand, to a degree not 
paralleled in other animals, that they are participating in a cooperative interaction involving a shared 
understanding of a situation. A human gestures with the knowledge that other humans understand 
the gesture’s meaning, and can use the gesture themselves for their own ends. Moreover, they know 
that the gesture is useful because other humans agree on its meaning. They can even participate with 
others to alter the gesture’s purpose to be “I’d like the sweet treat, not the salty one.” To humans, the 
gesture is not just a tool, or a means to an end that might happen to involve others. It is a movement 
that others could similarly use, or modify, because of its shared, conventional meaning. Therefore, 
it is the uniquely human recognition of symbols as conventional that allows us to construct new 
symbols, to change the meaning of existing symbols, and to situate symbols in the context of other 
symbols. These abilities allow us to flexibly communicate with others, to aid our own thinking, and 
to construct new ideas or revise old ones. Apes, on the other hand, have a comparatively reduced 
tendency to participate in established convention of what gestural movements come to mean, or 
ought to mean (Tomasello, 2019). Apes instead gesture because they want something, and they’ve 
associated the gesture with getting that thing; that is, they merely exploit, via learned association, 
what has already been established as convention by humans. 


So at least in humans, symbolic behaviour involves a cognitive capacity that pushes beyond mere 
association of action with outcome. It pushes beyond the association of some gesture, marking, or 
utterance with meaning. Symbolic behaviour instead exemplifies an active engagement with symbols 
as entities whose meaning is established by convention. The distinctions we draw here might seem 
unimportant, or merely a game of semantics. But the added burden of actively engaging with symbols 
in this way is a critical piece of the puzzle for designing AI that is as symbolically fluent as humans. 
Thus is true whether we imagine symbol-use in a social setting, or their use purely within an individual 
(e.g., to aid one’s reasoning, as Feynmann recalls doing when he invented new notation to study 
trigonometry; Feynman and Sackett, 1985). For example, do we want AI that can only form brute 


Symbolic Behaviour in Artificial Intelligence 


associations between scratches on a page and things in the world, or do we want AI that can conjure 
up a new symbol to aid its reasoning, as a mathematician would when exploring the frontiers of our 
knowledge? Do we want AI that instinctively performs some action when given a particular signal, or 
do we want AI that cooperates with us to devise new signs and signals so that our future interactions 
are more efficient and fruitful? And how would AI do the latter if it has no conceptual understanding 
of meaning being constructed through convention? How would it do the latter if it doesn’t know how 
to actively participate in an infrastructure of shared meaning? So, while we do not claim that the 
way we construe symbol use is “correct”, we nevertheless argue that it is better aligned with what we 
want symbolic fluency to look like (and what it looks like in humans), and hence, it is an important 
direction for AI research. 


To this end, we propose some interrelated behavioral traits that reveal an active participation in 
an infrastructure of meaning by convention. In particular, symbolic behaviour is receptive, constructive, 
embedded, malleable, separable, meaningful, and graded. We will unpack each of these in turn, and 
speak briefly to how they are being addressed (or how they might be addressed if they are not) in the 
current AI literature. 


Receptive. Symbolic behavior includes the ability to appreciate existing conventions, and to 
receive new ones. For example, humans can use a new word when its meaning is conveyed, while 
acknowledging that others also have this meaning in mind when they use the same word. This criterion 
might seem odd given the examples at the beginning of the section regarding dogs and apes, which de- 
emphasized the understanding of a symbol’s meaning for establishing evidence of symbolic thought. 
However, being receptive to an entity and its associated meaning is still a basic ability required 
for symbolic understanding, even if it might not constitute sufficient proof that one understands 
something as symbolic. As we’ve seen, many animals can associate sounds, gestures, words, or images 
with certain meanings through brute association, without an accompanying understanding that 
these meanings are a matter of convention. So, while being receptive to meaning is a basic mode of 
participation in a symbolic framework, it does not on its own entail fluent symbolic thought. 


Receptiveness to meaning is arguably the predominant behavioural trait explored in current AI 
research. For example, researchers assess language models by their ability to engage with existing 
convention (as measured by the quality of the text they produce, and whether it is meaningful to a 
human onlooker). Work on multi-modal perception measures engagement with existing convention 
by asking a model to convert between words and some of the things humans have in mind when they 
use those words (e.g., by producing an image of a scene given some textual description) (Radford 
et al., 2021, for a recent example). Instruction-following agents map between symbolic inputs and 
goals (Hermann et al., 2017). New research also shows that deep learning models can be receptive to 
established convention with only a single exposure (or a few exposures) to a new symbol (Brown 
et al., 2020; Hill et al., 2020). Evidently, the ability to engage with established convention manifests 
in a number of ways, and research in this space is currently thriving. 


Constructive. A second trait of symbolic behaviour is the ability to form new conventions by 
imposing new meaning on effectively arbitrary substrates. Whereas the previous behavioural trait 
(receptiveness) speaks to a system’s ability to appreciate the meaning of a symbol that is imposed by 
someone else, this property refers to the dual ability to create a new symbol when doing so would be 
useful. This ability is critical for communicating efficiently (e.g. creating a new term for a recurring 
situation) as well as for creating new systems of knowledge (much scientific progress begins with the 
creation of a new symbol, e.g. a “gene”). Indeed, one substantial benefit of engaging with symbols 
is that one can reduce the mental burden of thinking about a complex concept by denoting it with 
a symbol, even if the symbol is used purely internally. Constructive use of symbols demonstrates 
both the understanding that creating symbols is useful and understanding of the fact that, because 
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meaning is conventional, it can be imposed arbitrarily on top of any substrate. 


Evidence for this behavioural trait is scarce in current AI research. While much of what we do 
comprises creating models that engage with the conventions humans have already established, less 
work probes a model’s capacity to construct new conventions by imposing meaning on arbitrary 
substrates. Few-shot learning explores a model’s ability to rapidly assign meaning to a new substrate, 
but the meaning is usually still human-constructed (€.g., categories of images that are relevant to 
humans). Evidence for this behavioural trait might instead look like the following: (i) A model is 
asked a question, and when explaining its answer it invents a new symbol to streamline or clarify its 
reasoning process or communication. (ii) Agents cooperating on a team invent terminology so that 
they can communicate without exposing their strategy to their opponents. (iii) A model constructs 
an algebra and uses it to prove something meaningful, or similarly, we withhold knowledge of a 
particular mathematical concept and its associated symbol and see whether a model proposes a way 
to solve a problem that invokes that concept and symbol. 


Embedded. Symbols conform and contribute to the principles of organization that describe the 
broader symbolic system in which they’re associated, and symbolic behaviour reflects this understand- 
ing. Deacon again: 


“The learning problem associated with symbolic reference is a consequence of the fact 
that what determines the pairing between a symbol (like a word) and some object or 
event is not their probability of co-occurrence, but rather some complex function of the 
relationship that the symbol has to other symbols. . . symbols cannot be understood as an 
unstructured collection of tokens that map to a collection of referents because symbols 
don’t just represent things in the world, they also represent each other.” (Deacon, 1998) 


Because the meaning of a symbol is determined in part by how that symbol interacts with the rest 
of the symbol system, introducing new symbols can radically alter the interpretation of other symbols. 
For example, the discovery of category theory as a unifying perspective on many mathematical research 
areas fundamentally changed the questions researchers ask within many of those areas (Mac Lane, 
1988). 


There are a few ways that current AI models demonstrate behaviour that appreciates the embed- 
ded nature of symbols. Neural networks in particular have strong representational and functional 
priors that bias them toward such behaviour: since their internal vector representations can be 
compared in magnitude and angle, as any one representation changes so too does its relation to 
other representations. One way this manifests is with distributed word embeddings trained using 
large corpuses of human generated text, which capture semantic content like analogies (Mikolov 
et al., 2013). However, this way of hard-coding the embedded nature of symbols can sometimes 
have undesirable consequences, such as when symbols indiscriminately absorb the biases of the 
input data (Bolukbasi et al., 2016; Caliskan et al., 2017). While Lake and Baroni (2018) show that 
learning relationships among symbols in a restricted context can lead to poor generalization from 
deep learning models, these models may generalize better when language is embedded within a richer, 
more embodied context (Hill et al., 2019; see also McClelland et al., 2020). Indeed, humans can 
use embodied understanding, such as physical gestures, to help them grasp abstract mathematical or 
scientific concepts (Goldin-Meadow, 1999). Similarly, knowledge of underlying social constructs aids 
logical reasoning in analogues of the Wason Selection Task (JOHNSON-LAIRD et al., 1972; Sperber 
et al., 1995). The whole structure of knowledge in which symbols are embedded can, and usually 
does, affect symbol-use. 


Malleable. A symbolic thinker demonstrates an understanding that the conventionality of meaning 
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allows for change, and thus that the commitment to the current meaning of a symbol is not ironclad. 
This allows for meaning to be altered by the creation of other symbols or concepts, as well as the 
ability to integrate a redefinition of a word, or to be able to redefine a symbol oneself. 


Indeed, human language use exemplifies the malleability of a symbolic system, because our 
language use is highly collaborative and pragmatic (Clark, 1996; Clark and Wilkes-Gibbs, 1986; Frank 
and Goodman, 2012). The interpretation of language depends heavily on the situation in which it is 
used and the mutual understanding between the speaker and listener, and so understanding human 
communication requires an ability to form pragmatic conventions on the fly (Clark and Wilkes-Gibbs, 
1986; Hawkins et al., 2018). That is, using and responding to words and utterances in conventional 
and contextual ways, rather than fixed and unchanging ones, is necessary for achieving the flexible 
language use that humans exhibit.” 


But the malleability of symbols is not only important in everyday communication. Another benefit 
of understanding that symbol meaning is conventional is the consequent understanding that meaning 
could be otherwise, or even should be otherwise. This epistemic humility can be critical to insight, 
and many important human ideas could not have been obtained without expanding or altering the 
meaning of a symbol. For example, extending the concept of numbers to include complex numbers 
allowed humans to mathematically describe the wave nature of fundamental particles, and revising 
the geocentric concept of the universe allowed us to develop a heliocentric model of the solar system. 
To reason flexibly and productively, it is often necessary to reshape the conventions of meaning, 
when the extant meaning is misaligned with the physical world or with other aspects of the symbol 
system. But to do this, one must first know that meaning can be reshaped because it is established by 
convention. 


It’s not obvious whether any extant AI system exhibits behaviour demonstrative of a malleable 
understanding of symbols, at least compared to how this behaviour manifests in humans. While we 
previously provided evidence that current models can rapidly appreciate new convention, it’s less clear 
whether they exhibit behaviour that demonstrates that they can, or should, shift their understanding 
of an already known symbol. Returning to the case of language, consider a person from North America 
visiting Britain, who will quickly change their internal definitions of the words “quite”, “boot”, and 
“lift”. Any learning-based models will surely alter their understanding of a symbol as they experience 
new data; in other words, malleability is entailed by learning. However, this type of malleability is 
passive, and depends on researchers to drive it. Humans more actively expand or alter the meaning 
of symbols. That is, they are malleable for a reason, whether that reason is to permit more fluid 
communication, or to come to a deeper understanding of some phenomenon. 


Separable. A symbol user should be able to demonstrate partitioning of their understanding of a 
substrate and symbol, or multiple symbols that reuse the same substrate. For example, a maple leaf 
can be both an entity with inherent properties (e.g., leaves are light and carbon-based), and a symbol 
for something else (e.g. Canada). Crucially, the understanding of the leaf as a symbol is not affected 
by superficial changes to the leaf (e.g. if the leaf were protected with a fungus-resistant chemical, 
unbeknownst to the person using it as a symbol). Furthermore, a single symbol may have multiple 
meanings which are separable from each other. For example, the word “bank” in English can refer to 
a border of a river or to a financial institution. But the separability of meaning goes far beyond a 
word having multiple fixed meanings. We can readily receive or construct a new convention with 
an existing symbol, without changing our understanding of the extant denotations of that symbol. 
For example, if someone is introduced to the programming language Python, they can adopt the 


2We note that some might dispute at what stage context plays a role in human language understanding — for example 
whether context only constrains the referents of indexicals or the implicatures drawn (Lasersohn, 2012) — but this debate 
is not relevant to our behavioral claims. 
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convention that Python = programming language in future conversations with programmers, without 
changing the way they interpret the word python when talking to a veterinarian. 


Although contemporary language models show the ability to partition symbol meaning among 
known senses, for example disambiguating the word “bank” in context, their ability to fully separate 
meaning remains to be seen. In particular, can these models apply new meaning to a known word? 
For example, could these models handle challenges like “When I say ‘walk’ I mean ‘swim’ and when 
I say ‘park’ I mean lake. If I walk in the park, will I get wet?” Or can they handle a new use of a 
word created after they were trained? This will be important for models to adapt appropriately to 
shifting language across contexts and time. The ability to partition substrate properties from symbol 
properties will also need further investigation. 


Meaningful. Symbolic behavior should demonstrate use and communication of the meaning 
behind the symbols and reasoning processes that employ them. Consider what happens when we 
interpret symbol-use as consisting purely of syntactic manipulation. Harkening back to the symbol 
grounding problem (Harnad, 1990), in purely syntactic systems meaning must somehow propagate 
through meaning-agnostic syntactic reasoning processes. But meaning itself is essential to effective 
use of symbols. We don’t employ algebras because we get an arbitrary physical token at the other 
end, but rather because we can interpret the reasoning process and the resulting arbitrary physical 
tokens as meaningful. Fields medalist Paul Cohen highlighted this aspect of symbolic thinking in 
mathematics: “I never was able to successfully analyze proofs as a combinatorial ‘game’ played with 
symbols on paper” instead, to reason productively “one must essentially forget that all proofs are 
eventually transcribed in this formal language” (Cohen, 2002). Mac Lane (2012) explains why: 
“strict formalism can’t explain which of many formulas matter [...] the choice of form is determined 
by ideas and experience.” The meaning of symbols and their relationships gives mathematicians 
intuitions for why a theorem might be true, and thereby ideas of how to prove it (as do examples 
illustrating the theorem, see Polya, 1990). McClelland et al. (2016) make similar arguments about 
the primacy of meaning in mathematics. Similarly, Marder (2020) describes theory in biology as 
“disciplined dreaming” wherein we face the “challenge of creatively marrying the rules of mathematics 
and physics with what is known of fundamental biological principles.” It is meaning that solves the 
frame problem (Dennett, 2006). It is meaning that drives effective symbolic thinking. 


Meaning is not just useful for constraining reasoning, however. A major goal of theories or proofs 
is to convey understanding. Thus, mathematicians are often unsatisfied with a formal proof of a 
theorem that does not clearly communicate understanding of the underlying ideas (Lange, 2015; 
Mac Lane, 1988): 


“Mathematicians must search for the most illuminating proof. What matters is proofs 
which are “deep”— but not in a logistic sense of depth. The logical rules of inference 
provide a careful syntax for each proof, but they do not serve to indicate the steps that 
are crucial — those steps where one understands why the proof works. For this reason, 
Mathematics is not just logical deduction.” Mac Lane (1988) 


Models must therefore understand their own reasoning processes as meaningful to exhibit fully 
meaningful symbolic behaviour. Harnad alluded to this fact — despite focusing on grounding 
tokens, he noted that “the syntactic manipulations both actual and possible and the rules [must be] 
‘semantically interpretable” (Harnad, 1990). That is, the principles by which the model manipulates 
its symbols must themselves be understood symbolically by the system. Symbolic behaviour should 
reveal the construction of “deep” arguments, not merely chains of valid logical inferences that somehow 
arrive at the correct conclusion. 
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Evidence for meaningful understanding of symbols and reasoning will therefore come in the form 
of both being able to use semantics to reason more effectively (for example, finding a proof of a 
theorem efficiently rather than by blindly searching) as well as to communicate how the interaction 
between the semantics and the reasoning process licenses those conclusions (i.e. communicating 
why the semantics motivates particular steps of the proof, and what that implies about the concepts 
in question). A number of recent advances in AI have used deep learning to partially address the 
first point. For example, AlphaZero (Silver et al., 2017) effectively uses the learned semantics of its 
estimates of move value as strong heuristics for reducing the search space of possible actions. However, 
because AlphaZero’s reasoning process (Monte-Carlo Tree Search) is hand engineered and relies 
on strictly rule-based processes, it is incapable of achieving the second goal. This motivates future 
research toward systems that can understand their reasoning processes as meaningful. 


Graded. Finally, symbol use is not a binary capacity (c.f. McClelland et al., 2010, 2016). We’ve 
presented behavioural criteria that collectively demonstrate symbolic fluency, and each encapsulates 
a spectrum of competencies. As a simple example, young children might be receptive to existing 
language conventions, but only simple words and phrases. The reasons why a child is not receptive to 
some words or phrases might have to do with seemingly tangential cognitive faculties. For example, 
perhaps the child is inexperienced with parsing certain sounds, and hence has difficulty perceiving a 
word’s aural boundary. Or perhaps they cannot reproduce the word vocally, and by virtue of having 
little experience using the word, they have difficulty appreciating its meaning. One can imagine a 
number of other scenarios involving the limitations of memory, attention, and nearly every other 
cognitive faculty. What this example further illustrates is that symbolic thinking is not an independent, 
siloed function, but is rather an interdependent component of our broader cognition. And since other 
aspects of our cognition—attention, memory, perception—can be graded, so too can our ability to 
think symbolically. 


For example, although we highlighted the importance of malleability for allowing the revision of 
systems of knowledge, this does not mean that every human will be able to come up with the right 
alterations. Only a few may have access to the particular combination of information and experience 
that allow insight in any particular context, for example to realize that the heliocentric model of the 
solar system was more parsimonious than the geocentric one. Similarly, one can imagine inventing 
new conventions, but not with unbounded freedom. For example, consider the Piraha people of the 
Amazon who do not have words for exact numbers, and instead refer to imprecise quantities such as 
“small amount” or “larger amount” (Everett et al., 2005). While we might not expect them to create 
a symbol that meant “precisely five trees”, they clearly are symbolic thinkers, as evidenced e.g. by 
their use of language. Symbol use will always be constrained in one way or another (by cognitive, 
conceptual, or cultural forces) and hence will inevitably present as a graded capacity. 


As it regards AI, acknowledging that the ability to use symbols is graded entails acknowledging 
that symbol use is not synonymous with a particular computation (e.g. syntactic control over entities), 
or the use of a particular type of representation (e.g. discrete tokens). We must instead ask how 
certain computations or representations alter the behaviour of a system, and use analyses of the other 
criteria presented above to characterize how a system expresses engagement with symbols. Symbolic 
behavior manifests in a variety of ways in humans, and it’s reasonable to expect that it will manifest 
in a number of ways in AI as well. 


3.1. What Symbolic Behaviour is Not 


When motivating the physical symbol system hypothesis, Newell and Simon made reference to the 
development of formal logic as a “symbol game”, stating that: 
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“Logic, and by incorporation all of mathematics, was a game played with meaningless 
tokens according to certain purely syntactic rules. All meaning had been purged. One 
had a mechanical, though permissive (we would now say nondeterministic), system about 
which various things could be proved. Thus progress was first made by walking away 
from all that seemed relevant to meaning and human symbols. We could call this the 
stage of formal symbol manipulation.” (Newell and Simon, 1976) 


This view dominated our conception of symbols in AI. Symbols became synonymous with discrete 
entities, symbolic processing was what we did when pushing symbols around with formal algebras, 
and the properties that came to be associated with symbolic systems (e.g., compositionality) were 
inherited from these algebras. 


But it is because we are already symbolically fluent that we find value in purely mechanical, syntac- 
tic operations. As we noted above, mathematicians generally rely on meaning in their reasoning about 
which manipulations to make. Similarly, as we mentioned in regards to Shannon’s communication 
theory, purely mechanical computations are motivated by a deep, extant understanding of symbols. 
They're motivated by the knowledge that human interpreters can bring meaning back into the fold 
once the syntactic manipulation is complete. But crucially, purely syntactic operations per se are not 
synonymous with symbol-use because they do not consider the steps where meaning is extracted 
from, and later imbued in arbitrary physical tokens, which we argue is where a large part of the 
intelligence behind symbolic thinking lies. Claiming that AI is symbolically fluent because we equip it 
with syntactic-control of discrete entities therefore puts the cart before the horse; it assumes that 
such manipulations would be useful to a system that has yet to demonstrate an understanding of 
symbols as symbols, or worse, it assumes that symbolic fluency consists only of an ability to exhibit 
syntactic control of discrete entities. 


Notably, the importance of establishing symbol meaning was known by Newell and Simon, who 
wrote that “it is a hypothesis that these symbols are in fact the same symbols that we humans have 
and use everyday of our lives” (Newell, 1980). This has become an underappreciated, but critical 
assumption underlying GOFAI-like methods—namely, that it is possible to assume away the difficult 
piece of the symbolic puzzle: what meaning is, where it comes from, and who it is known to. To be 
fair, at the time this wasn’t an outlandish hypothesis, because if GOFAI-like systems indeed proved 
capable of feats of intelligence rivalling humans, and if human intelligence were proved to be best 
modeled by these kinds of symbolic methods, then the hypothesis would be corroborated. And at the 
time, no other class of model was as promising. But our discussion here highlights a key reason to 
suspect that this hypothesis does not hold: it’s not obvious how GOFAI-like models could interpret the 
entities over which they reason to be symbols, in order to exhibit the behavioral abilities that we have 
outlined. There is little evidence of these capabilities in GOFAI models, nor is there evidence that the 
meanings of the entities they manipulate are shared with humans (Harnad, 1990; Smith, 2019). 


Many contemporary approaches to symbolic computation—including hybrid architectures com- 
bining GOFAI-like methods with deep learning—take a syntax-centric view of symbolic processing, 
and hence do not advocate for symbolic behaviour as we have delineated it. Instead, they advocate a 
particular set of computational mechanisms that operate over entities that are symbols to the humans 
that create the systems. Crucially missing is a solution to how these things actually come to be symbols 
to the systems themselves. The only known proof-of-principle systems capable of symbolic thought 
are biological neural networks, and we have highlighted how many aspects of symbolic behavior can 
already be observed to some degree in contemporary neural network models. This suggests that the 
key to bringing symbols into the fold in hybrid systems might be their neural network “front-ends” 
and that, ironically, their GOFAI-like “back-ends” might be fundamentally preventing them from 
achieving human-like symbolic behavior. 
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Symbolic behaviour does not have to depend on rules. Rule-based reasoning is a useful tool, but like 
purely syntactic reasoning it is most useful when it is guided by a deeper capacity for symbolic thought, 
including the meaningful understanding of the reasoning process. That is, it is necessary to understand 
the meaning of the manipulations, and the relationship of the manipulations to the ultimate goal. 
Even in the most formal domains humanity has created, such as mathematics, rules are subsidiary 
to understanding. We noted above that meaning is often needed to guide mathematicians toward a 
proof, and to identify “deep” proofs that communicate the reason why the theorem is true. Symbolic 
behavior should likewise enable meaningfully constructed threads of reasoning that communicate 
understanding, which is notably not observed in syntax-centric GOFAI-like methods (Harnad, 1990). 


These matters come to the fore when thinking about important issues of safety and ethics. While 
it might be tempting to think that rule based systems could guarantee that AI will be safe or ethical, 
we suggest an alternative perspective, echoing others who have highlighted the challenges of creating 
rule-based machine ethics (e.g. Allen et al., 2005; Brundage, 2014). Applying strict rules will be 
problematic in domains of ethics and safety for reasons similar to those we have outlined in our other 
arguments: rule-based systems are incapable of understanding the meaning behind the rules, so they 
would only follow the letter of the law, not the spirit (the consequences of which were popularly 
fictionalized by Asimov in his Robots series of stories; Asimov, 1950). For example, we could imagine 
imposing a rule like “never hurt a human,” but in fact humans hurt other humans frequently in ways 
that we would consider ethical. What if you need to break a bone to set it cleanly, or amputate a 
gangrenous limb before it sickens the rest of the body? Or what if we have to tell someone about the 
death of a parent? Should we tell them right away? What if we need to withhold the news until they 
are in a safe situation to process it, not driving? Our ethical principles are profoundly contextual. 


Similarly, we might have a rule like “don’t discriminate on the basis of race,” but as practices such 
as redlining (in the US) have illustrated, it’s easy to find a proxy variable for race, like neighborhood, 
and have essentially the same discriminatory effect without directly using the protected category 
(Woods, 2012). We need a system that behaves in accordance with the meaning behind its principles, 
not one that just superficially follows the letter of the law. We need systems with judgement (Smith, 
2019). We suggest that this goal will be more easily achieved by pursuing AI that can exhibit symbolic 
behavior within a holistic, meaningful framework of ethics. 


4. How Can Symbolic Behaviour Come About? 


Symbolic behaviour can be characterized as an engagement with an infrastructure of meaning 
established by convention. In the last section we described the characteristics of this behaviour. Here 
we hope to answer: what are the cognitive steps a system must take to form or appreciate an agreed 
upon link between a substrate and what it (arbitrarily) means? Our contention is that a necessary 
(but not necessarily sufficient) cognitive trait is the ability to reconcile perspectives between oneself 
and others, and that this trait may emerge as a consequence of effective and efficient social and 
cultural engagement. Further, we’ll argue that the emergence of symbolic behavior (McClelland 
et al., 2010) will be most effectively induced through external social and cultural forces, even in 
systems that primarily engage in introspective symbol use. This perspective is certainly not unique, 
and has appeared in a number of guises in the literature. For example, specifically in the context of 
language, many researchers have considered the role of successful communication as a bottleneck for 
learning the use of symbols (Kirby, 2002; Lazaridou and Baroni, 2020). Nevertheless, our perspective 
considers the more general case of symbol use (language or otherwise). And as we will describe, it 
highlights some cognitive prerequisites for symbol-use, rather than suggest algorithmic features (such 
as “bottlenecking”, compression, efficiency, etc.) that might necessarily entail symbol-use. 
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One strategy to elicit an engagement with an infrastructure of meaning by convention in AI is 
to first identify the types of inferences that humans make when engaged in symbol-use, and then 
target these same inferences when building AI. These inferences have a clear developmental ontogeny. 
When children are around two years old, they certainly use words as tools. They also come to learn 
that seemingly arbitrary behaviours, such as nodding their head, have meaning. But at around three 
years old, children show some evidence of inferring—and then obeying—social norms based on 
others behaviour (Schmidt et al., 2016). At around three years old, children also begin to exhibit 
an appreciation of language as a matter of convention. They assume that words will be known by 
speakers of the same language (but not others), they expect speakers of their language to use words in 
accordance with what they conventionally mean, they reinterpret or ignore noun-object bindings if a 
speaker is unreliable later, and (if they are bilingual) they mix languages less (Dautriche et al., 2020; 
Tomasello, 2019; Wei, 2020). At a similar age, they begin to show signs of considering what could be 
in the mind of others—what they might know or how they might intend to use a word (Diesendruck, 
2005). As they age further, their ability to understand meaning as a matter of convention solidifies. 


Tomasello argues that effective social engagements, like those listed above, require the ability 
to recognize distinct perspectives of different social members, and to take an objective perspective 
which subsumes them. He states: 


“A chimpanzee sees a monkey escaping, and he knows that his conspecific sitting next 
to him sees the monkey escaping also...They are both attending to the monkey escaping, 
and each knows that the other is too. But they are not jointly attending to it; they are not 
attending to it as a “we”. Two humans in that same situation could, if so motivated, attend 
to the monkey escaping together in joint attention. This creates between the two of them a 
kind of shared world, within which they each distinguish their two perspectives. They each 
also understand that both of their perspectives—that is, their beliefs—on the situation 
could potentially contrast with an objective (perspectiveless) view of it.” (Tomasello, 
2019) 


Humans construct cognitive representations that do not merely reflect a passive, subjective 
experience. Rather, these representations strive towards objectivity; they reflect coordination across 
subjective perspectives and what exists in the world, and hence come to be about things that are 
true from multiple perspectives, and perhaps even regardless of perspective. For example, suppose I 
come in from outside and start acting as though it’s raining outside. Perhaps I collect my umbrella 
and put on a rain coat so that I do not get wet when I go out again. But you recently looked out the 
window, and saw that it was not raining. You must then reconcile and coordinate our perspectives. 
But crucially, you coordinate under the pretense that there is an objective truth to the matter; it is 
either raining or it is not regardless of what one of us might think. One of our subjective perspectives 
is mistaken, and perhaps it’s your own because in the time since you looked out the window it could 
have started raining. You therefore come to believe it is raining, despite your subjective experience. 


The capacity to achieve these coordinated, relatively objective perspectives can emerge as a 
result of optimizing social “tasks” (cf. Graziano and Kastner, 2011). Consider the classic false-belief 
paradigm, wherein your colleague observes an object being moved from one container to another 
without your awareness. If you then search for the object in the wrong container and ask for your 
colleague’s help, they will not help by continuing to search the incorrect container. Rather, they will 
immediately go to the new container, because they understand that your perspective conflicts with the 
objective state of the world. There is no way you could know the object moved, whereas your colleague 
can because they perceived it, and you must reconcile their behaviour (and hence, their perspective on 
the state of the world) with your perspective on the state of the world. Thus, a successful interaction, 
wherein you retrieve the object from the proper location, can fuel the cognitive machinery necessary 
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for making the types of inferences involved in reconciling various subjective beliefs. And since we 
have a notion of “success”, we can imagine a situation wherein it is optimized for across experiences. 
One can imagine a glut of other situations that can drive the cognitive development required for 
objective understanding when optimized, such as jointly attending to a situation or sharing intentions 
when performing a task. For example, children draw upon reconciling perspectives in many situations 
to understand what to learn from a social interaction (Gweon, 2020), and even what to teach to 
others — for example, they will choose to teach things that would be more difficult to discover or 
more valuable to know (Bridgers et al., 2020). Teaching and learning could be both intrinsically and 
extrinsically rewarding for children, which could encourage them to improve their ability to reconcile 
perspectives in order to achieve these rewards. 


Understanding the objective “truth” of a situation by reconciling subjective perspectives might 
seem unrelated to understanding convention. However, the skills required for reconciling perspectives 
are similar to — and may well be prerequisite for — those required for understanding conventions. 
Consider again what the understanding of objective perspectives requires: one must reconcile subjec- 
tive perspectives in the context of an external world that provides the “ground truth”. Understanding 
and creating convention, on the other hand, requires reconciling subjective perspectives without 
necessarily being able to appeal to, or depend on any “ground truth”. In particular regards to symbols, 
understanding a symbol’s meaning cannot come from understanding the objective properties of the 
symbol’s substrate, which do not convey meaning in and of themselves by definition. Symbol meaning 
is instead inherently subjective, and often heavily depends on a reconciliation and coordination of 
subjective perspectives alone. 


While we’re calling attention to an extreme case here (i.e., reconciliation of subjective perspectives 
without any hope of appealing to some source of objective truth), how things play out in the real 
world is naturally more graded. For example, consider children’s first experiences with symbols, such 
as the word “dog”. The people with whom they communicate are predictable users of the word “dog”, 
and can be counted on as dependable sources to ground the word (and hence, give it meaning). 
This renders the meaning of the word “dog” nearly as “objective” as any visual, tactile, or auditory 
experiences of actual dogs. But it’s important to note that in these cases meaning is grounded in 
things in the world that are not the symbol per se. The symbol substrate’s properties are not doing 
any work to convey meaning, and initially, it’s only via brute association that children come to learn 
that an essentially arbitrary substrate (air vibrations) is being used to refer to something else entirely 
(a dog). In particular, noun learning in children comprises a gentle learning path to the important 
meta-lessons that: (1) seemingly arbitrary things can be connected €.g., certain air vibrations and 
dogs), (2) there are established connections that other people in their environment adhere to. 


Nevertheless, children eventually cope with situations where they can’t depend on “objective” 
things in the world to understand meaning. For example, they encounter metaphoric uses of language, 
they participate in analogical reasoning without immediate access to relevant aspects of the physical 
world (e.g., when told that “a zebra is like a horse with stripes” without having ever seen a zebra; 
(Harnad, 1990)), they learn words that do not have clear connections to the physical world (such as 
“wish” or “no”), and they might even participate in the invention of a new word when playing a game. 
These experiences require coordination of subjective perspectives between themselves and the people 
with whom they’re communicating, without being able to depend on a source of objective truth of 
the matter (which does not yet exist). The shift in cognitive abilities is indeed reflected in human 
development: children first absorb established convention from their social and cultural context before 
they develop the fluency to be active participants, and creators of new convention (Tomasello, 2019). 
In many ways, reconciling subjective perspectives is harder when we cannot appeal to an objective 
state of the world as an arbiter, which could explain why it appears later in cognitive development, 
why early child language contains more words for concrete entities than adult language (Brown, 


14 


Symbolic Behaviour in Artificial Intelligence 


1973), and why it is a skill that is most pronounced in humans compared to other animals. 


A promising path forward for developing AI capable of symbolic thought, then, comprises cooper- 
ative interaction with the goal of shared understanding. AI systems must help each other understand 
meaning when there is no clear objective arbiter to which they can appeal. While this may be a 
difficult thing to engineer, there is a silver lining: cooperative interaction is desirable on its own, so 
we can reap its benefits if it also happens to be conducive to building AI capable of symbolic thought. 


Some work in AI has begun to make inroads in optimizing exactly these types of interactions at 
scale (Abramson et al., 2020, e.g.), specifically in the context of human-agent interactions, as opposed 
to multi-agent interactions alone. We see this as a promising direction. While we can imagine the 
capacity for symbolic behaviour would emerge eventually from multi-agent interactions, we believe 
that human-agent interactions will provide greater pressure to achieve symbolic behaviour. As noted 
above, human communication demonstrates many flexible aspects of symbolic behaviour—humans 
expect their collocutors to reconcile perspectives and reason pragmatically—so an agent learning to 
communicate effectively with humans in context must learn these abilities. Furthermore, learning from 
human interactions is important if one of the end goals of AI is to develop systems that cooperate with 
humans, because these interactions will allow agents to bootstrap from the rich cultural knowledge 
that humans have already accumulated, just as new generations of humans do (Tennie et al., 2009). 
Learning from human interactions will therefore allow agents to learn what we think is important 
about the world, and how to communicate with us about it, and ultimately how to construct new 
knowledge beyond what is known. There is much to be explored at the interface of AI and social 
learning (Vélez and Gweon, 2020). 


4.1. A Purely Internal Origin for Symbolic Thought? 


While we’ve emphasized the ability to reconcile subjective experiences between individuals in some 
social setting as a path towards symbolic fluency, there is an alternative possibility. Perhaps this ability 
originates internally, and just happens to be exploited in social settings. If this were true, then the 
capacity for understanding convention in social settings would be a cognitive spandrel (Gould and 
Lewontin, 1979). That is, the ability to engage with social conventions would be a consequence of 
internal machinery that originated from other demands. We will discuss this possibility in this section. 
Note, however, that the following discussion is not about how a system that is already symbolically 
fluent can and does use symbols of their own creation, outside of any social context. As we noted above, 
symbolically fluent systems undoubtedly use these abilities for internal reasoning as well as external 
behaviours. What we ask here is how this ability to use symbols—introspective or otherwise—can come 
about in the first place. Can the ability to use symbols emerge entirely within an individual? 


Many researchers have answered this question in the positive. Some have gone as far as to posit 
that “symbol manipulation” is an innate cognitive capacity that is a prerequisite for acquiring language 
and other human abilities. For example, Fodor argued that humans must have an innate (not learned) 
internal language of thought that possesses the characteristics of classical symbolic methods (Fodor, 
1975). Similar arguments were historically advanced by Chomsky about the need for a Universal 
Grammar in order to learn language (Chomsky, 1995), although this view has been extensively 
critiqued (e.g. Dabrowska, 2015) and Chomsky and colleagues have more recently suggested that 
recursion alone might suffice (Hauser et al., 2002). It is unclear what AI can learn from disputed 
notions of innateness—what should we build in and why? The onus is on us researchers to describe 
and demonstrate how any mechanisms we propose lead to the behavioral characteristics highlighted 
here. 


We can, however, entertain an “internal” origin story of symbolic fluency without succumbing to 
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the problems of innate machinery. Consider the case where a system still must reconcile perspectives, 
but not between itself and another. Rather, it must reconcile the perspectives of its past and current 
self. Therefore, reconciliation and coordination of internal perspectives might provide similar sorts of 
cognitive pressures to those that exist in social situations. Indeed, Karmiloff-Smith (1994) posited 
a process of representational redescription in cognitive development that could involve a kind of 
internal reconciliation in order to obtain a more objective representation. Thus, it is conceivable that 
an isolated agent could eventually discover self-symbolic thinking, without social pressures to drive it. 
Of course, this origin story of symbolic thought comes with its own difficulties. For example, a system 
must have an impeccable memory in order to store the perspectives it must eventually coordinate, 
and it must have reasons for reconciling them that induce the same kinds of pressures as those that 
exist in socio-cultural settings. 


If symbolic fluency can be a consequence of purely internal processes or events—whether mech- 
anisms per se, or forces like those described in the previous paragraph—then we’d expect to see 
evidence of this in nature. We might expect, for example, that humans exhibit symbolic fluency from 
a very young age. And if this ability were evolved, they might also exhibit symbolic fluency (in the 
absence of social pressures) with great ease. We believe the evidence for this is scarce, however. 
While humans do show fascinating behavioural characteristics from a relatively early age, such as 
the ability to learn and generalize abstract patterns from only positive examples (Marcus et al., 
1999), these abilities do not require a classical symbol manipulation mechanism. Instead, they can be 
demonstrated in simple neural network models given appropriate learning experience (Geiger et al., 
2020). Furthermore, learning formal domains like mathematics is rather difficult for humans: even 
by the end of a proof-based high school geometry class most students do not achieve formal deductive 
understanding of the concepts taught unless (or until) they become undergraduate mathematics 
majors (Burger and Shaughnessy, 1986). Similar patterns have been observed in linguistics. Gleitman 
and Gleitman (1970) showed that only graduate students (compared to undergraduates or those with 
a high-school degree) exhibited a strong and consistent ability to understand three word compound 
nouns, despite compound nouns being a relatively basic construction that should be easily parsable 
according to the syntactic rules of language. If social engagement is a secondary consideration for 
the development of symbolic fluency, why would our ability to understand formal symbolic domains 
emerge so much later than our ability to engage socially? We suggest that this provides some evidence 
for social pressures as primary, although it is by no means conclusive. 


The above evidence is inconclusive in part because it is challenging to accurately measure “internal” 
symbolic fluency, especially in an experiment that is not confounded with social engagement. It is 
challenging to design a test that can only be passed if one were to have the capacity to use symbols, 
because alternative solutions are always possible. For example, although symbolic thinking was surely 
essential to humanity achieving artificial flight, evolution was able to discover flight as well, and we 
would not want to attribute symbolic thinking to evolution. So while we cannot rule out the possibility 
of self-symbolic thinking arising without a social context, it would be very difficult to ascertain. 


Indeed, the difficulty of measuring internally-developed use of symbols is itself an argument 
for exploring social forces as a tool for teaching AI to exhibit symbolic behaviour. The social forces 
highlighted by Tomasello and others provide directions for training systems to engage with structures 
of conventional understanding (Tomasello, 2019). It seems necessary for a system to develop symbolic 
behaviour in order to fully engage in these social situations. By contrast, for the same reasons that it 
is difficult to assess an internally-originating use of symbols, it is correspondingly difficult to imagine 
situations that would ensure the emergence of an internally-originating use of symbols rather than 
some alternative solution. Thus, we suggest that AI research should focus more on interactive social 
situations, especially interacting with humans, because this will provide more direct pressure toward 
developing symbolic behaviour. 
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5. Reconciling Our View With The Current Debate 


Much of the current and past debate regarding symbolic AI and connectionism revolves around the 
particularities and properties of certain ways of computing. For example, strengths and weaknesses 
of neural networks and GOFAI-like methods are contrasted and compared, and a reconciliation of the 
two is presented as a remedy to Al’s current woes (Garcez and Lamb, 2020; Garnelo and Shanahan, 
2019; Lake et al., 2017; Marcus, 2018, 2020). Our argument, on the other hand, takes a decidedly 
behavioural perspective, with no explicit mention or prescription of the functions or mechanisms 
required for symbolic thought to emerge. This approach has a number of advantages (c.f. McClelland 
et al., 2010). First, it allows us to explicitly characterize what we expect from a system that uses 
symbols. Notably, we go beyond base mechanistic capacities (such as “variable binding”), which do 
not engage with what symbols are in the most general sense, but rather focus on what we humans 
tend to do with symbols in the particular setting of digital computers or formal algebras. Second, it 
leads us to the important conclusion that symbolic behaviour is a graded capacity that is not suddenly 
enabled if a system comes equipped with a certain type of computational mechanism. Third, it better 
aligns with symbolic behaviour as observed in nature, where it has a developmental ontogeny and is 
graded based on experience, maturation, and formal education. 


The evidence cited in arguments for GOFAI-like mechanisms in humans is often behavioral. For 
example, humans show a capacity for relatively systematic or compositional generalization. This 
capacity is often assumed to rely on particular GOFAI-like compositional mechanisms (e.g. Fodor and 
Pylyshyn, 1988), which have these properties by definition. Our views differ from that perspective in 
several crucial ways (see also Hill et al., 2019; Potts, 2019). First, under our interpretation symbol-use 
does not entail systematicity since it emphasizes the graded, flawed nature of behaviour, rather than 
something lost in the translation of some “purer”, internal formal algebra that has these properties by 
definition. Indeed, even adult humans are often far from perfectly systematic, achieving compositional 
generalization performance of only 70-90% even within relatively simple domains (Lake et al., 2019).° 
Moreover, symbolic frameworks need not be strictly compositional, even if some, like formal logic, 
are compositional by design. Natural language, for example, is a symbolic system that isn’t strictly 
compositional (Fodor, 2001; Szabó, 2020), and it’s trivial to invent a symbolic system that is not 
compositional. The view we’ve presented embraces this lesson, and does not conflate symbols or 
symbolic frameworks with possible properties those frameworks can have. This view has an important 
consequence: if symbolic behaviour is not inherently systematic, then mechanisms that guarantee 
systematicity (through, for example, built-in compositional computations) may be neither necessary 
nor sufficient for achieving the type of symbol-use we advocate for here. Rather, systematic behavior 
may be a graded competency afforded by environmental and educational factors (Hill et al., 2019; 
McClelland et al., 2010, 2020). 


Many perspectives on symbolic AI implicitly define symbols differently than we do in this work, 
often synonymizing them with discrete tokens or even localist representations (Garcez and Lamb, 
2020, e.g.). That view meshes well with the types of entities we operate with using digital computers, 
and has allowed us to leverage the algorithms and mechanisms developed for digital computers 
when designing AI. But what is discrete about air vibrations, or the fluctuating intensities of photons 
reflected off a cloth waving in the wind? Humans undoubtedly have an ability to interpret what we’d 
traditionally not call discrete entities as symbolic, even if they also interpret them as discrete units 
with a broader symbolic framework (the important distinction here is that a particular entity does not 
have to itself be discrete for it to be a member of a discrete set, much like how a continuous vector 


3This imperfection is often elided in claims about differences between humans and models, as noted by Lampinen 
(2020): “When humans make a compositionally-valid interpretation 80% or 95% of the time, it is presented as evidence of 
their compositional symbolic skill, yet when a deep model achieves 98.4% accuracy on much more difficult [mathematical 
reasoning] problems, it is depicted as a failure of the model class to exhibit compositional symbolic reasoning.” 
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can be a member of a discrete set of basis vectors). Why would we not expect the same for the AI we 
create? If we are to appreciate symbolic thought in its most potent, general form, then we need to 
embrace the idea that symbols come in many shapes and forms. Indeed, according to the definition 
presented here, any substrate can be symbolic, and a fluent symbolic thinker should appreciate and 
engage with this fact. 


Some recent work has attempted to explain how symbols can come to be within more classical 
frameworks, for example by representing symbols through mechanisms like combinatoric logic 
(Piantadosi, 2020), or deriving symbolic abstractions through probabilistic program induction (Ellis 
et al., 2020). While we find these directions intriguing, we are not optimistic that they will ultimately 
provide a path to the symbolic behavior we have outlined. These systems are sometimes capable of 
constructing new abstractions, though not in a manner that demonstrates a command over convention. 
It is less clear whether they can achieve other attributes of symbolic behaviour that we highlighted 
above. In particular, it is not easy for these systems to have malleable symbols in the way that we have 
described, as the definition of a symbol is still generally expressed as a single program or formula. 
That is, it may be difficult for these systems to accommodate the many subtle constraints that we 
argue influence human interpretation of e.g. a word in context. Furthermore, it is still unclear how 
these systems could demonstrate their understanding of their reasoning as meaningful, and as we 
suggested above, it is the meaning behind reasoning that solves the frame problem. Thus, these 
systems, like the symbolic mechanisms of GOFAI, are prone to combinatorial explosions of possible 
inferences. For example, even though Ellis et al. (2020) use deep learning to guide program search, 
their approach is still restricted to relatively simple domains with tightly constrained underlying 
structure. It is thus still unclear whether these approaches are a pathway toward human-like symbolic 
fluency, especially in real-world settings. However, as we noted above, our arguments are relatively 
agnostic to the particular mechanisms underlying behavior. Our arguments about the power of social 
interactions for encouraging symbolic behavior should apply to these mechanisms as well. Thus, 
we suggest that those interested in such models should consider how they could accommodate the 
richness of human social interaction. 


6. Summary & Conclusion 


A symbol consists of some substrate that is imbued with meaning by matter of convention. Crucially, 
a substrate is only a symbol to those who demonstrably: (i) understand the established conventional 
meaning, and (ii) understand that meaning is what it is because of convention. 


To determine whether a system understands something as a symbol, we require evidence that 
it understands that it is actively participating in an infrastructure of meaning by convention. In 
the biological world, this evidence presents as knowledge that some meaning is a matter of social 
norm, and as a capacity to establish new conventions and update old ones. To this end, we define 
some behavioural characteristics that demonstrate symbolic fluency: symbolic behaviour is receptive, 
constructive, embedded, malleable, separable, meaningful, and graded. 


We can imagine two possible origins of symbolic behaviour that are not mutually exclusive: 
symbolic behaviour as an emergent consequence of social engagement, and symbolic behaviour as a 
consequence of internal forces. Although we do not hope to fully adjudicate which course led to the 
human development of symbolic behaviour (and some evidence points to a co-evolution; Deacon, 
1998), we have argued that considering the social setting might be the most promising pathway 
toward developing AI capable using symbols in a human-like way. 


The views we’ve presented here seem to run counter to the past and current AI zeitgeist surrounding 
symbols, which emphasizes rigid mechanisms and particular representational forms. Nevertheless, it 
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would be difficult to create a symbolically fluent AI without carefully characterizing the behaviours 
that result from the use of symbols. We need AI that is driven by a deep understanding of symbols as 
members of an infrastructure of meaning by convention, which manifests as behaviour that is nuanced, 
flexible, and subject to various shaping forces. We therefore prescribe that AI practitioners focus 
on instilling symbolic behavior in AI if they hope to build something that engages as profoundly as 
humans do with issues of ethics, reasoning, and even beauty. 
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